Uncertainty-Aware Self-Training for Low-Resource Neural Sequence Labeling

Neural sequence labeling (NSL) aims at assigning labels for input language tokens, which covers a broad range of applications, such as named entity recognition (NER) and slot filling, etc. However, the satisfying results achieved by traditional supervised-based approaches heavily depend on large amounts human annotation data, may not be feasible in real-world scenarios due to data privacy computation efficiency issues. This paper presents SeqUST, novel uncertain-aware self-training framework NSL address labeled scarcity issue effectively utilize unlabeled data. Specifically, we incorporate Monte Carlo (MC) dropout Bayesian neural network (BNN) perform uncertainty estimation token level then select reliable tokens from based model confidence certainty. A well-designed masked task with noise-robust loss supports robust training, suppress problem noisy pseudo labels. In addition, develop Gaussian-based consistency regularization technique further improve robustness Gaussian-distributed perturbed representations. alleviates over-fitting dilemma originating pseudo-labeled augmented Extensive experiments over six benchmarks demonstrate that our SeqUST improves performance self-training, consistently outperforms strong baselines margin low-resource scenarios.